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I. Introduction 

The Vocational Education Act of 1963 (VEA 63) greatly 
enlarged the role of publicly supported occupational edu- 
cation in the United States, Since 1963, occupational 
educational curricula have proliferated, and now encompass 
an. extensive list of occupations. The student population 
served by occupational education has increased and become 
more heterogeneous • 

According to VEA 63, occupational education must be 
provided to "... those in high school, those who have com- 
pleted or discontinued their formal education and are pre- 
paring to enter the labor market, those who have already 
entered the labor market but need to upgrade their skills 
or learn new ones, those with special educational handicaps, 
and those in post-secondary schools,"* Thus one of the 
principal objectives of contemporary occupational education 
is to create genuine vocational alternatives for many 
segments of the population. To fulfill this objective in 
New York State, occupational curricula are offered at a 



* From Sec* 101 of the Vocational Education Act of 1963 as 
amended by the Vocational Education Amendments of 1968, 
Public Law 90-^576. 
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variety of educational centers including conununity colleges, 
secondary schools, and Boards of Cooperative Educational 
Services . 

The growth of occupational education in structure, scope 
and enrollment has been accompanied by increased accountabili 
requirements. Regional administrators of occupational edu- 
cation are required by law to synthesize a variety of in- 
formation related to the occupational programs offered in 
their region and the extent to which different student groups 
are served, and to submit this information to state and 
federal education authorities. In addition, local school 
boards, regional employers and other local interest groups 
have exhibited rising concern about the quality of occu- 
pational program offerings. Consequently, it has become 
increasingly important to evaluate the effectiveness of 
occupational program offerings, and to base program planning 
and fending decisions upon such evaluations. 



F/142-5-00 



II. Reporting and Evaluation. 
Requirements for Occupational Education 

To satisfy the need for more effective reporting and 
evaluation, the New York State Education Department decided 
to develop a Reporting and Evaluation System for Occupational 
Education.* Riverside Research Institute (RRI), in coordination 
with the New York State Education Department Division of 
Occupational Education Planning, carried out several tasks 
which have resulted in a general design for this system. 
Reporting and evaluation requirements for occupational edu- 
cation, and the tasks carried out by Riverside to arrive at 
the system design, are discussed in this section of the report. 
A. Reporting Requirements for Occupational Education 

1 , Mandated requirements . 

Under federal and state mandates, a substantial 
amount of information about occupational education must be 



* The development of a Reporting and Evaluation System for 
Occupational Education is one of three major tasks carried 
out under contract #C-504 8 4 between the New York State Edu- 
cation Department and Riverside Research Institute • The 
other tasks carried out by RRI under this contract are des-- 
cribed in The Coordination of Program Planning and E valuation 
Systems for Occupational Educatioiv i Volume II: A Targeting 
S\stem for Occupational Education ." Volume III: A Monitoring 
System for Occupational Education . 
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collected by the New York State Education Department. School 
districts, Boards of Cooperative Educational Services (BOCES) , 
public colleges, the Big Six schools and ether agencies are 
required to file reports which are used to monitor the ex- 
penditure of state and federal funds, to develop the State 
Plan for occupational education (required under VEA 63) , and 
to assess the extent to which occupational programs carried 
out at various educational centers are in conformance with 
that plan. For each program administered, local educational 
authorities are required to provide enrollment information, 
including the numbers of each type of student (general, 
disadvantaged, handicapped) enrolled in the program, the 
numbers of students enrolled at various educational levels 
(grades 9-12, adult supplementary, adult preparatory, etc.), 
and the ethnicity of the students . 

In addition, the State Education Department (SED) re- 
quires follow-up reports for each program offered. Follow- 
up reports contain the number of students completing and 
net completing each program, and a variety of other infor- 
mation concerning the characteristics, education and em- 
ployment status of the program's participants. Information 
which describes the programs, such as program budgets, 
personnel characteristics, types of program offerings and 
so on, is also submitted to the SED* 
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2 . Additional reporting requirements . 

In addition to formal reports required by state 
and federal authorities, occupational education data must be 
provided for regional public relations, planning and many 
other purposes. Specific information on the status of 
occupational education is often desired by administrators of 
local school districts, school boards, regional occupational 
education advisory committees, union officials, parent- 

9 

teacher associations, and minority group representatives. 
Reporting of this kind is essential to the successful ad- 
ministration of publicly supported occupational education. 

Reports to parent and teacher groups, school boards, 
students and others support guidance functions, familiarize 
individuals with regional occupational program offerings and 
interpret the expanded role of occupational education to the 
community. Reports to the regional advisory committees in 
various occupational fields are an important part of the 
program planning process. Reports to employers, union of- 
ficials and industry representatives serve to facilitate 
cooperative education programs and to create a job market 
for occupational program graduates. 

B . Evaluation Requirements for Occupational Education 
1. Assessment of program ef fectiveness^ 

Follow-up studies are the traditional means of 
assessing the effectiveness of occupational programs. If a 

F/142-5-00 - 5 - 



substantial proportion of a program's graduates obtain jobs 
or advanced placement in the occupational cluster for which 
they were trained, it is assumed that the program is ef- 
fective. Traditionally, program effectiveness has been 
thought to be a matter of the contribution that the program 
makes to the employment status of its graduates. In the 
context of Riverside's studies, this definition of effective- 
ness has been associated with the effectiveness of the occu- 
pational educational system as a whole,* Program effective- 
ness is taken to mean the degree to which the specific, 
stated objectives of occupational programs have been fulfilled. 

Thus follov7-up evaluations do not measure program 
effectiveness, i.e., they do not allow the occupational edu- 
cator to determine whether or not programs have succeeded in 
transmitting the skills and knowledge that they are designed 
to impart. Recently, attempts have been made to establish 
explicit curriculum objectives related to the entry-level 
requirements of the occupations for which training is being 
provided. If the students meet the objectives, the program 
is considered effective. If a large number of students do 
not meet the objectives, it becomes necessary to find out 
why and to revise the program accordingly. 



* See Volume II of the final report for a discussion of system 
effectiveness . 
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2 • Evaluation for occupational education planning # 

A capability to evaluate programs is needed for a 
variety of planning purposes within occupational education. 
The planning of specific courses, for example, should be 
based on an evaluation of previous results. Thus, occu- 
pational educators need to be able to assess the consequences 
of different instructional methodologies on program costs 
and performance outcomes. A variety of important variables, 
such as different levels of teacher training, can affect 
program effectiveness. These variables must be identified 
so that effective programs can be designed, and the effective- 
ness of existing programs improved. 

An evaluation capability will also be helpful to 
iaentify t:he student abilities and knowledge which are neces- 
sary for satisfactory performance in different programs. 
Such information will allow meaningful entry requirements to 
be developed for occupational programs* 
C. Existing Reporting and Evaluation Procedures 

Not all of the reporting and evaluation requirements 
previously discussed are currently being met within occu- 
pational education . Follow-up studies constitute the only 
form of evaluation which is consistently undertaken and 
the quality of these studies is extremely variable. Reporting 
requirements are often met with guesses rather than with 
real data and reports are therefore frequently invalid. 

F/142-5-00 - 7 - 
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D. T asks Undertaken for the Development of a Reporting and 
Evaluation System for Occupational Education 
1* The determination of informational requirements 

and functional specifications for the Reporting 

and Evaluation System . 

In order to develop a set of functional specifica- 
tions for the Reporting and Evaluation System, both the kinds 
and amounts of information that the system would require were 
investigated. In addition to student names and addresses, 
program enrollments, school district identifications, ethni- 
city and other typical information elements, the requirements 
for the inclusion of additional student characteristics (e.g., 
objective test scores, cognitive ability, socio-economic 
status) and new effectiveness indicators (in both the cognitive 
and atfective domains) were studied. The results of these 
efforts and the functional specifications of the system which 
were obtained are reported in Section III A of this report. 
2. The development of a general design for the 

Reporting and Evaluation Sysbem . 

On the basis of the determination of informational 
requirements and functional specifications, alternative ap- 
proaches to a reporting and evaluation system were analyzed, 
and the System was designed. The System description is given 
in Section III B of this report. A description of ho\^ the 
System meets the functional specifications is given in Section 
III C. 

F/142-5-00 - 8 - 
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3 . Software requirements for the Reporting and 
Evaluation System , 

The development of the Reporting and Evaluation 
System included the specification of some of the required 
electronic data processing software* Appropriate procedures 
were developed for storing student, program, and personnel 
information in a computer, and for the organization of data 
banks to be used for longitudinal (e.g*, time-series) 
evaluations* In addition, the applicability of Riverside's 
software packa-^je, System for the Evaluation of Educational 
Data (SEED)^ to the Reporting and Evaluation System was 
established, the package was modified, and a SEED user's 
manual was written. The suitability of SEED for the 
Reporting and Evaluation System is described in Section 
III D. The SEED user's manual is presented in Appendix I. 

4 • Field trial of the Reporting and Evaluation System . 

The system design was empirically tested during 
the contract period at the Nassau County Board of Cooperative 
Educational Services (BOCES) center. This field trial is 
described in Section IV of this report* 

5 . Statewide implementation strateg y^ 

On the basis of the field trial, a strategy for 
statewide implementation of the Reporting and Evaluation 
System was developed jointly with State Education Department 
personnel. This strategy is described in Section V. 
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III. Design of a Reporting and Evaluation 


1 


- 


System for Occupational Education 






A* Determining the Functional Specifications of the 






Reporting and Evaluation System 






1. Preliminary system study. 






In the initial phase of the work, RRI met with 






members of the State Education Department Division of Occu- 






pational Education Planning to define current and projected 






reporting and evaluation requirements for occupational 






education. 






Following the definition of these requirements. 






RRI conducted a system study of reporting and evaluation 






operations and procedures at the educational centers where 






occupational programs are offered. The purposes of this 






study were: 


J 

1 




• to identify current and projected needs for 


-J 

•* 




reporting and evaluation at the local level; 






• to assess difficulties which have arisen in 






connection with meeting current reporting and 






evaluation requirements; and 






9 to determine how a statewide reporting and 


• 




evaluation system could be implemented and 






sustained at the local level. 
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The system study was primarily conducted at the 
Nassau County Board of Cooperative Educational Services 
(BOCES).* The study was conducted at a BOCRS for several 
reasons : 

• the BOCES are conducting an increasing portion 
of the secondary and adult occupational pro- 
grams offered in New York State; 

• in many important respects, reporting processes 
at the BOCES are similar to reporting processes 
at Big-Six and post-secondary agencies; and 

o since the State Education Department requires 
the BOCES to assume substantial reporting 
responsibilities for their own programs as 
well as the occupational programs offered with- 
in the school districts in each BOCES region, 
BOCES reporting problems include the reporting 
problems and requirements of local school 
districts . 



* In addition to the system study carried out at the Nassau 
County BOCES, RRI obtained additional information concerning 
local reporting and evaluation operations by accepting an 
additional task. RRI carried out a computer synthesis of 
Vocational Education Data System (VEDS) 1970-71 reports, sub- 
mitted to the State Education Department from community 
colleges, BOCES, and school districts throughout the State. 
This task provided insight into reporting difficulties and 
into procedures for eliminating them. 
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The system study ^revealed that a variety of prob- 
lems exist in connection with the execution of existing 
reporting and evaluation functions, and also identified many 
reporting and evaluation requirements which cannot be met 
at present. For example, BOCES centers are expected to 
obtain information about the numbers and different types 
of students enrolled in occupational programs offered at 
all local educational centers within each BOCES region. 
Local district personnel are asked to survey students in 
these programs and to transmit the required information to 
the BOCES center. 

In practice, conflicts arise between BOCES and 
local district personnel because, in order to obtain the 
required information, teachers or guidance counselors in 
the local districts must be diverted from other tasks or 
asked to complete the surveys as an additional task . Neither 
of these choices is an attractive one, and it is not sur- 
prising that the BOCES have difficulty in obtaining cooperation 
from local districts . 

Furthermore, the BOCES are dependent upon the 
local districts to purchase services provided by BOCES centers. 
Not all local districts do so, and the BOCES are interested 
in increasing the number of local districts which directly 
support their operations. Under the circumstances, the BOCES 
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are reluctant to exert pressure on either affiliated or non- 
affiliated local districts to complete student surveys for 
occupational educational reporting and evaluation purposes. 
2. Functional specifications of the System , 

On the basis of the system study and the earlier 
conferences with State Education Department personnel, a set 
of functional specifications for the Reporting and Evaluation 
System were formulated. 

a. The System must be able to meet all legal 
reporting requirements applicable to occupational education 
(see Section II A.l). 

These requirements have been imposed by the 
State Education Department Division of Occupational Education 
Planning to provide information for the State Plan and for 
the preparation of the statewide annual report on occupational 
education . 

The System must be able to provide summaries 
and other required information for regional, "state of occu- 
pational education" reports (see Section II A 2). 

Such reports are needed to inform school 
districts, regional constituencies, school boards, potential 
employers, labor unions, advisory groups, and others, of the 
status of occupational education in each region. For the 
purpose of creating a job market for occupational graduates, 
all administrative units providing occupational education 

F/142-5-00 - 13 



(e.g., local school districts, community colleges, Big-Six 
cities, BOCES) need the capability to produce such reports. 

The BOCES have additional regional reporting 
requirements — particularly for secondary students--which must 
be met by the System. Some local school districts send 
secondary students to BOCES centers to receive occupational 
education, and pay tuition to the BOCES for these students. 
The BOCES are accountable to these school districts, and 
therefore must have the capability to provide individual 
student information, performance summaries for each student 
sent by each district, and other information of interest to 
the local districts. 



ations of program effectiveness. 

Since the results of follow-^up studies are 
used to assess the effectiveness of occupation^:..', programs, 
the Reporting and Evaluation System must be capable of sup- 
porting the execution of follow-up studies for program 
graduates as well as students who do not complete programs. 
The System should be able to carry out follow-up studies 
independently or m coordination with the Targeting System. 



of program effectiveness with respect to a variety of new 
effectiveness criteria. For example, a relatively recent 



c . 



The System must be able to carry out evalu- 



The System must also permit the evaluation 



* 



See Volume II of the final report. 
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development in assessing the effectiveness of occupational 
programs has involved the establishment of explicit behavioral 
objectives and procedures for determining whether or not program 
objectives have been met. The Reporting and Evaluation System 
must be capable of incorporating such data, whatever its form, 
without revision of the System. In addition, the System should 
be capable of prov.iding summary analyses of the extent to which 
skills and knowledge have been imparted in particular programs. 
This capability would enable regional and statewide planning 
objectives to be formulated in terms of the delivery of spe-- 
cific skills and training to students, rather than in terms 
of the services provided and the number of students expected 
to participate in programs. 

In addition to assessments of program ef- 
fectiveness, the System must be capable of identifying the 
conditions under which programs are effective and the con- 
ditions under which they are not. For example, the Reportiiig 
and Evaluation System must have the capability to include and 
analyze the following student characteristics of recognized 
importance in occupational education: measures of cognitive 
functioning; measures of occupational interests; aspirations; 
measures of attitudes toward school and occupational education; 
and demographic characteristics such as age, sex, race and 
socioeconomic status . 
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The System must also be capable of incorpo- 



rating specific measures which may be needed for particular 
programs: medical information, measures of eye-^hand co- 
ordination, manual dexterity, and so on. In the past, occu- 
pational educational administrators have collected at least 
some of these student data, but have not had the means to 
assess their impact upon program effectiveness. In addition 
to student data, the System must be capable of processing 
teacher characteristics and other information relevant to 
the assessment of program effectiveness. 



should provide the capability to assess the effectiveness of 
programs relative to costs. To be capable of carrying out 
cost--benef it analyses, the System must be able to process 
program cost data and to analyze these data with respect to 
a variety of effectiveness measures, including the indicators 
derived from system effectiveness studies. 



keeping requirements for student certification. 

In occupational education, it is possible to 
set rational behavioral program objectives and to determine 
whether students have met them. The objectives can be con- 
sistent with the entry level skills required for the jobs 



Finally, the Reporting and Evaluation System 



d. 



The System must be capable of meeting record- 



★ 



See Volume II of the final report. 
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for which the programs are providing training. For example, 
behavioral objectives suitable for a program in refrigeration 
equipment maintenance could include the ability of students 
to dismantle and reassemble refrigerators of different types. 

When students have met the objectives, they 
may be certified as having the training and skills necessary 
for work within a given occupational category. Some form 
of statewide certification would undoubtedly help to create 
a job market for program graduates, and to improve training 
standards within occupational education. Riverside's system 
study, conferences with SED personnel, and studies of the 
occupational educational literature indicate that certification 
of competence in a given field is both feasible and desirable. 
Therefore, Riverside recommends that procedures for generating 
certificates of competence in particular occupations be included 
in the Reporting and Evaluation System. 

The Reporting and Evaluation System should 
be capable of being the record-keeper of student performance. 
Since some programs incorporate courses which are taken by 
students in different years, the System must be capable of 
keeping records over long periods of time. Finally, the 
Reporting and Evaluation System must be able to process 
student records with certification algorithms developed by 
occupational educators, and to report which students are 
entitled to certification in each program. 
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The System must be capable of test scoring 



and analysis. 



Test scoring and analysis are required for 



the reporting, evaluation and certification functions. 



f • 



The System must be capable of attendance 



processing, course scheduling, and the execution of a variety 
of fiscal functions within occupational education. 



fiscal operations are not completely germane to reporting and 
evaluation as defined in this report, many of these operations 
make use of data that would be contained and processed in the 
Reporting and Evaluation System. Redundancy could be elimi- 
nated by linking these operations to the Reporting and Evalu- 
ation System. 

3. Structural specifications of the System. 



usable by the diversity of educational centers which offer 
occupational programs in New York State. These units include 
two- and four-year colleges, Big^Six agencies, school districts 
which subscribe to BOCES occupational services, and those 
which do not. 



Educational Data System (BEDS) and other information systems 
under development or currently maintained by the State Edu- 
cation Department. 



Although attendance, course scheduling and 



The Reporting and Evaluation System must be 



b. 



The System must be compatible with the Basic 
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c. The System must serve to reduce the time 
required of personnel in local educational centers to fulfill 
reporting and evaluation requirements, 

3. Description of the Reporting and Evaluation System 
1 • General system flow . 

Figure 1 is a schematic representation of the 
Reporting and Evaluation System. As indicated in the figure, 
the System includes operations carried out at or near the 
sites where occupational programs are offered (Regionalized 
Occupational Education Operating Functions) , as wall as 
operations performed by a Centralized System Integration and 
Management Unit. For illustrative purposes. Figure 1 shows 
only one local occupational educational unit (e.g., a school 
district, a BOCES center, or a community college) linked to 
the Centralized System Integration and Management Unit. In 
actual practice, the Integration and Management Unit would 
have responsibility for all local occupational educational 
units . 

As indicated in Figure 1, local operating functions 
such as attendance, enrollment, test scoring, and grade re-^ 
porting are integrated at the local level with the support of 
local or regional data processing centers. Thus the data 
associated with the local operating functions would be checked, 
organized, and aggregated (when necessary) at the local level. 
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Management Control would develop forms^ schedules, guidelines 
and procedures for local information collection and integration 
These materials would be developed to ensure that needed 
information is efficiently gathered and to ensure tnat the 
information requirements of the State Education Department, 
local occupational educators and other groups are being 
fulfilled. When necessary to do so. Management Control 
would also develop software to be used locally for the 
efficient integration of local operating functions. 

Data would be sent by the local data processing 
units to the System Integration and Management Unit to be 
organized and added to a central, computer data base. A 
computer software package, System for the Evaluation of 
Educational Data (SEED) , would be used by Management Control 
for data maintenance purposes. Management Control would 
have the responsibility of organizing data collection processes 
and meeting requests for reports and evaluations from the 
State Education Department, from regional educational centers, 
and from all other sources* 



Specific procedures for integrating local operating functions 
might vary to some extent with the type of local occupational 
educational agency administering the programs (e.g., BOCES, 
community college, Big-Six city, etc.). However, it is RRI's 
judgment that relatively standard pr':ir:tices can be and need 
to be developed for the execution of local operating functions. 
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2 . The data base » 

The largest component of the data base would contain 
student information. Information for each student receiving 
occupational education would be maintained within a computer 
data base. As indicated in Figure 1, the information would 
be stored in matrix form. Each student occupies a row of the 
matrix, and each student datum is stored in a particular 
column of the matrix. 

The entry of information into the student data 
base begins at the time of enrollment of the student in an 
occupational program. Management Control would provide guide- 
lines and forms to help guidance counselors, admissions 
personnel and others involved in the local enrollment process 
to obtain student information for subsequent entry into the 
student data base. Student information to be entered would 
include student name. Social Security number, address, race, 
sex, ethnicity, educational level (secondary, post-secondary, 
etc.), type of student (handicapped, disadvantaged, regular), 
parent's occupation, available objective achievement data, 
and any other information (such as medical data) which can 
be obtained conveniently when students enroll. The Office 



The measurement of social immobility will require additional 
information, such as each student's family members, friends in 
the region, attitudes, and so on. See Volume II of the final 
report . 
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of Education Program Code corresponding to the program that the 
Student was taking would also be included. 

The student's Social Security number is very 
important to the Reporting and Evaluation System. It pro- 
vides a unique numeric identifier for each student while 
avoiding the problems associated with maintaining a separate 
identification system. Although many occupational education 
students currently do not have Social Security numbers, they 
would be asked to obtain them. Assistance can be provided 
trom local educational agencies for this purpose. 

As the student continues in occupational education, 
more information is added to the data base. For example, 
Management Control would issue computer lists containing 
student names, occupational education course numbers, and 
Social Security numbers. These lists would be used by teachers 
to record students' grades. The lists would then be returned 
to the central data processing unit, where grade information 
would be entered into the student data base. 

Similar procedures would be used to enter test 
scores, aggregated attendance data and other information re- 
quired for evaluation and reporting purposes. The last elements 

Office of Education Program codes may be found in Vocational 
Education and Occupations (19 6 8) . 

The Social Security number has considerable advantages for 
the System. The Social Security Administration will assign 
numbers to students if requested to do so. In addition, there 
is evidence that, within a few years, the Social Security 
Administration will automatically assign numbers t.o elementary 
school children . 
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of information to be entered would probably be student place- 
ment and follow-up data. 

The student data base would be supplemented in some 
instances by other information. For example, another component 
of the data base would contain information about teaching 
personnel. This component would contain Social Security num- 
bers, experience, training, type of teacher (regular, para- 
professional) , teaching assignment (course code) , salary 
information and other data. For the Big-Six and large local 
occupational educational agencies, a third component of the 
data base would be organized by occupational program. 
C. Meeting the Functional Specifications. System Outputs 

and their Utilization . 

1. Legal reporting requirements . 

a. Enrollment reports . Enrollment reports would 
be generated by Management Control by aggregating information 
contained in the student data base. The information could be 
aggregated to meet the needs of the State Education Department 
Division of Occupational Education Planning. For example, for 
each occupational program given at each local center, the 
number of enrolled students would be reported. This informa- 
tion would be sorted by student sex, ethnicity, educational 
level and type of student (general, disadvantaged or handi- 
capped) . For enrollment reports as well as other information 
required by the State, data would be submitted from each local 
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occupational unit to the Integration and System Management 
Unit. There it would be stored, aggregated over units according 
to State Education Department needs, and submitted to the State 
Education Department in a convenient form, 

b. Follow-up reports . The Reporting and Evaluation 
System could produce follow-up reports in two ways. On the 
assumption that the Targeting System is operating. Management 
Control would retrieve (from the student data base) the Social 
Security numbers of students who completed each program as well 
as those who did not. The Socidl Security numbers would then 
be traced in the Targeting System data base, the Dictionary 
of Occupational Titles (DOT) codes for the graduates' jobs 
would be obtained, and the relatedness of the graduates' jobs 
to their training would be assessed.* 

If the Targeting System were not operating, the 
Reporting and Evaluation System would carry out follow-up 
studies in the following manner. Lists of addresses and 
telephone numbers of students who had completed programs, as 
well as those who had not, would be obtained from the student 
data base. Management Control would produce computer-generated 
mailing labels and send our a carefully designed follow-up 
survey form on which student Social Security numbers had been 
pre-entered. A number of questions would be asked to determine 

* This procedure is described in Volume II of the final report. 
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the employnient status of each student being surveyed. As the 
surveys were returned, the data would be collected by the local 
occupational units and subsequently entered into the student 
data base* For sLudents who did not respond, lists of student 
names, last-known addresses. Social Security numbers, and last- 
known telephone numbers would be sent to the local occupational 
units. Students would be telephoned by local personnel and the 
follow-up data would be entered in the lists, VJhen a sufficient 
number of students (or their parents) had been reached, the 
lists would be returned to the System Integration and Manage- 
ment Unit, where the new information would be merged into the 
student data base. 

Management Control would then aggregate the 
information in the student data base to fulfill the State's 
requirement for follow-up information. As in the case of 
enrollment reports, follow-up data could be aggregated to meet 
current planning and evaluation requirements of the State 
Education Department, For example, the numbers of students 
who completed and the numbers of students who did not complete 
each program within each local occupational unit would be 
reported. The information could be partitioned by sex, 
ethnicity, type of student, educational level, and employment 
status (unemployed, employment related to training, employment 
not related to training, part time, full time, continuing 
education at higher level, etc). 
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c. Personnel data , Personnel information would 
be reported to the State Education Department by Management 
Control. The information would be obtained and aggregated 
from the personnel file. In accord with current reporting 
requirements, the following personnel data could easily be 
reported for each local occupational unit: the number of full- 
time teachers, part-time teachers, paraprof essionals , and so 
on, teaching in various occupational fields. 

2 . Other reporting requirements ^ 

Management Control would meet a variety of reporting 
requests that come from local occupational educational units 
and other sources. Management Control would formalize a 
request system so that individual administrators could have 
indirect access to centrally stored data. Since it is not 
possible at this stage of system development to predict all 
of the ways in which occupational administrators might want 
data analyzed and reported for regional purposes, it is not 
possible to describe the exact procedures for meeting these 
regional reporting requirements. However, SEED software 
(described in Section III D) will enable Management Control 
to fulfill virtually any request for aggregation or analysis 
of information which had been stored previously in the data bas( 

3 . Evaluations of program effectiveness . 

As indicated elsewhere in this report, there is a 
definite trend in occupational education to establish explicit 
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program objectives. These objectives are sometimes termed 
"behavioral objectives" or "career educational performance 
goals." The objectives are based upon the entry-level skills 
required in the jobs for which the program is providing training 
Since the objectives may be related to required entry-level 
skills as defined by job analysts, the objectives are easier 
to develop in occupational education than in practically any 
other educational area.* 

For programs which have explicit objectives, informa- 
tion may be collected concerning the extent to which each 
student has met the objectives. This information, coupled 
with grades and other student data, may be stored within the 
student data base of the Reporting and Evaluation System and 
used as the basis for evaluations of prcgram effectiveness.** 

At the end of a program, a full summary of the extent 
to which various student groups have met the program's objec- 
tives would be reported to local occupational education adminis- 
trators. These data would also be aggregated into a general 

* For example, an objective for a figure clerk might be as 
follows: when given a bundle of 100 cash slips fastened by a 
rubber band, a rubber thumb, and a 10-key adding machine, the 
student must be able to obtain a tape total from the adding 
machine. A second total will then be run for verification. 
One tape will be dated and attached to the bundle of sales 
slips. The entire operation is to be completed within five 
minutes (Huffman & Byers, 1971) . 

** Such information could also be used for program monitoring 
processes. Program monitoring in occupational education is 
described in Volume III of the final report. 
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program effectiveness summary- If costs could be associated 
with programs, then costs-effectiveness measures would also 
be computed and reported. 

If a program was judged to be unsuccessful in pro- 
viding the skills and knowledge required to enter the occupa- 
tional cluster to which the program was directed* (i-e,, when 
students did not meet the program's objectives), local occupa- 
tional educators could request evaluative analyses from cen- 
tralized management to help to identify any program deficiencies. 

Virtually any type of evaluative analysis could be 
carried out. For example, the importance of meeting the 
objectives could be determined by analyzing the relationship 
between the attainment of objectives and follow-up employment 
experiences. Through the statistical analysis of program 
results, it would also be possible to identify factors which 
are correlated with failure to meet program objectives (e.g., 
student reading level, student attitudes toward occupational 
education, attendance) , The results of such program analyses 
would be given to local occupational educators along with an 
interpretive summary and a set of recommendations for adjusting 
the program. On the basis of such reports, occupational educa- 
tors could plan a variety of changes in the program to improve 
its effectiveness. For example, occupational educational 
personnel could request local districts to devote more resources 

* See Volume II of the final report for a discussion of job 
similarity and job relatedness. 
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to improving reading skills, or request them to introduce 
activities designed to motivate students to attend relevant 
courses . 

The Reporting and Evaluation System has been designed 
to fulfill many evaluation requirements. Programs with similar 
objectives but using different pedagogical approaches could be 
compared to determine which approach is most effective for 
differerxt student groups. The effects of an array of factors 
(such as class size, teacher experience, student occupational 
aspirations, attitudes toward school) upon program success 
could be determined. Studies of dropouts from occupational 
education could be conducted* The evaluative processes which 
the System is capable of performing are far too numerous to 
mention in these pages. However, the system will provide the 
capability for general program evaluation as well as the capa- 
bility to carry out idiosyncratic evaluations of particular 
programs for a variety of local purposes. 
4 . Student certification . 

Once program behavioral objectives have been establish! 
for occupational programs, and once procedures have been applied 
to objectively assess whether students have met the objectives, 
it will then be possible to construct a meaningful certifica- 
tion system. When students have met the objectives of each 
course within a program, they ma^ be certified as having mas- 
tered the entry-level skills of corresponding occupations. 
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Since many programs extend over more than one year, the Reporting 
and Evaluation System student data base would be used to store 
student data for any number of years. As each student finished 
each course, the extent to which behavioral objectives were 
met would be added to the data base. 

The criteria for certification will vary from program 
to program. In some cases, occupational educators might think 
it necessary to meet all program objectives. In other cases, 
some fraction of the total number of objectives might be con- 
sidered sufficient for certification. In still other instances, 
it might be considered important for students to meet specified 
critical objectives plus a fraction of all other objectives. 

All certification criteria would be incorporated by 
Management Control into certification algorithms. These 
algorithms would operate on the data in the student data base. 
Lists of students who had met the certification criteria would 
be generated. Information could also be provided on students 
v/ho did not meet the certification criteria, and the objectives 
which they failed to fulfill to attain certification could be 
identified. 

Since follow-up data would be included in the student 
data base, it would be possible to assess the effects of certi- 
fication on the employment status of various groups of occupa- 
tional graduates. Thus, the Reporting and Evaluation System 
could be used not only to maintain a statewide certification 
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system, but also to perform some needed evaluations of student 
certification requirements.* 

5, Test scoring and analysis. 

Test scoring would be carried out at the local level 
with standard test scoring software (provided, where necessary, 
by Management Control) • Test scores would be entered and 
maintained in the student data base. 

6 • Attendanc e , scheduling, census, grade reporting, 
and other administrative functions. 

Much administrative processing would be carried out 
at the local level with the support of Management Control. 
Since the results of much of the local processing would be 
merged into central data banks. Management Control would be 
active in providing local occupational education units with 
software, procedures, forms and other products which would give 
them the capability to carry out local processing efficiently. 
D . Management of the Reporting and Evaluation System 
Data Base. SEED 

!• The trade-off between efficiency and flexibility in 
computer software . 

A frequent obstruction in the development and instal- 
lation of management information systems is the substantial 
cost associated with writing and testing computer software. 

* The Targeting System (see Volume II of the final report) will 
also be useful in making such evaluations. 
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When a system cannot be completely specified at the outset, but 
must evolve over time, software costs can be devastating. Soft- 
ware developed on an ad hoc basis to fulfill initial specifica- 
tions often has to be re-written many times as the system 
specifications change and expand with system use. 

In the Reporting and Evaluation System for Occupational 
Education, specific system processes will undoubtedly be changed 
from time to time. Issues of accountability, reporting and 
evaluation have become important relatively recently in occupa- 
tional education, and thus there is still disagreement among 
occupational educators concerning ultimate reporting and evalu- 
ation requirements. Furthermore, federal and state reporting 
requirements are likely to change, the objectives in the State 
Plan will change, and, as indicated elsewhere in this report, 
program evaluation criteria will certainly change. 

If the Reporting and Evaluation System is to fulfill 
its purposes, the System must be able to assimilate these 
changes without major revisions in computer software. Flexible 
software must be used which will not require major modification 
as system requirements change and expand. However, such soft- 
ware will be less efficient than software specifically written 
to fulfill a particular purpose (i.e., more coirputer time will 
be required) . Highly efficient software is appropriate when 
system specifications are likely to remain stable for many years 
and only a minimal number of non-standard system applications 
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are expected to be carried out. For the Reporting and Evalua- 
tion System, the costs saved by applying efficient roftware now 
would be off-set several times over by continual software re- 
development in the near future. 

Thus, one of the tasks undertaken by RRI was an 
assessment of the applicability of an existing flexible soft- 
ware package. System for the Evaluation of Educational Data 
(SEED) , to the Reporting and Evaluation System. 

2 . SEED and its general suitability for occupational 

education , 

SEED is a flexible software system designed for the 
maintenance and analysis of educational data. During the past 
year, RRI has assessed the suitability of SEED for the Reporting 
and Evaluation System for Occupational Education, The general 
structure and capabilities of SEED were found to be suitable, 
although some changes and additions were necessary to adjust 
SEED to the specific needs of occupational education as defined 
in the system study. Some of these changes and additions were 
undertaken during the contract period and are reported in the 
next section. In addition, a nontechnical user's manual was 
written for SEED. This manual describes the application of SEED 
to data maintenance and analysis. The manual appears as Appen- 
dix A of this volume of the final report. The manual fully 
explains SEED'S capabilities and operating procedures. 
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SEED is capable of maintaining the data base described 
in earlier sections of this report. The specific data main- 
tenance capabilities of SEED include flexible data entry and 
data checking, data storage, merging new information, retrieval 
of individual student records, and the capacity to handle incom- 
plete data. 

SEED is capable of summarizing and reporting the 
data it maintains in virtually any manner. For example, once 
individual student and personnel data are entered into the 
data base maintained by SEED, the aggregation of individual 
data to fulfill current state reporting requirements takes just 
a few SEED English syntax commands. Reports generated by SEED 
may be in several forms: frequency distributions; cross- 
tabulations; plots of various types; and lists of particular 
groups of students by program, ethnicity, educational background, 
or any other characteristic included in the data bank. Since 
data in SEED-generated reports are labeled by SEED according 
to the user's directions, the reports should be easily intel- 
ligible to the recipients. 

In addition to SEED'S capacity to meet both state 
and regional reporting requirements, SEED'S analytic capabili- 
ties will allow for the fulfillment of the evaluation require- 
ments. SEED is capable of processing and summarizing follow- 
up data. Furthermore, grades, test results, attitude data, 
measures of the extent to which students have met behavioral 
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objectives, and virtually any other type of data relevant to 
the evaluation of occupational programs may be maintained and 
processed by SEED, SEED'S capabilities for providing statisti- 
cal analysis of data include: analyses of variance and covari- 
ance; t-tests; various correlations (biserial , tetrachoric , 
Pearson product moment) ; multiple regression analysis; multiple 
discriminant analysis; factor analysis; chi-square analysis; 
and analysis of uncertainty. 

SEED may also be used to maintain a student certifi- 
cation system for occupational education. Management Control 
would provide SEED with certification algorithms and English 
syntax instructions for each occupational program. SEED 
could then generate the names, addresses and Social Security 
numbers of students who were able to meet the certification 
criteria as well as those who were not. 

SEED software is open-ended. The user may attach 
temporary or permanent subprograms for certain types of analyses 
and reports. For example, since student names, addresses, and 
grades are maintained within the student data base, a subpro- 
gram may be added to SEED to generate students* transcripts 
and prepare them for mailing. 

3 • Modifications of SEED to make it suitable bo the 

needs of the Reporting and Evaluation System ^ 

Although SEED is generally suitable for the Reporting 
and Evaluation System, certain technical changes and additions 
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were desirable* These changes and additions were identified 
during the contract year, and the work efforts associated with 
the following changes and additions were undertaken: 

• SEED formerly accepted only 8 place numeric identi- 
fiers* Since the Social Security number is 9 places 
(and longer identifiers might possibly be needed) , 
SEED was revised to accept 16 place identifiers; 

• during the contract year, more analytic capability 
was added to SEED through the addition of capabili- 
ties for multiple discriminant and uncertainty 
analyses; 

• seed's cross-tabulation capability was further gen- 
eralized, and the output format was made more read- 
able to enable occupational educators without techni- 
cal experience to be able to interpret SEED cross- 
tabulation outputs; 

• the development of more efficient sort-merge software 
was begun during the contract period* Since student, 
personnel and program data will be put into the Re- 
porting and Evaluation System data base over periods 
of many months (and possibly many years for follow-up 
analyses, certification procedures and time-series 
studies), RRI determined that SEED'S sorting and merg 
ing capabilities were not sufficient for the large 
number of sorting and merging operations which the 
Reporting and Evaluation System would have to perform 
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It is Riverside's judgment that ^ few additional 
changes in SEED would facilitate its application to evaluation 
and reporting requirements. In addition to slightly increased 
analytic capability, and some changes in data maintenance pro- 
cedures to obtain more efficiency, SEED also needs a built-in 
capability to limit data base access to authorized users. RRI 
expects to execute these SEED changes and additions during the 
forthcoming contract year. 

4. The availability of SEED to the State Education 

Department . 

SEED is a proprietary system developed under RRI 
auspices. SEED is based upon a software package known as 
P-Stat which has been developed primarily by Mr. Roald Buhler, 
formerly the Director of the Princeton University Computer Cen- 
ter. Under an agreement with RRI, Mr. Buhler has given permis- 
sion for the development of SEED and has assisted RRI in this 
development. 

Since SEED is eminently suited to the reporting and 
evaluation requirements of occupational education, and since 
the further development of SEED was undertaken in part under 
contract-supported work efforts, RRI is prepared to provide 
SEED to the State Education Department without fee for its own 
use. RRI has requested that SEED be used only by the New York 
State Education Department, only within the boundaries of New 
York State, and that appropriate measures be taken to prevent 
the use of SEED by unauthorized persons. 
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IV. Limited Field Test of the Reporting 
and Evaluation System 

To test the feasibility of the Reporting and Evaluation 
System, a limited field trial was conducted at the Nassau 
County Board of Cooperative Educational Services (Nassau 
BOCES). Since administrators at the Nassau BOCES were pri- 
marily concerned with the fulfillment ol state reporting 
requirements, the field trial focused upon the reporting 
component of the Reporting and Evaluation System.* Nassau 
BOCES and RRI personnel coordinated work efforts to carry out 
the field trial with the support of personnel within the 
State Education Department Division of Occupational Education 
Planning. The field trial is summarized below. 
A. State-Required Program Enrollment Information 

Boards of Cooperative Educational Services have the 
responsibility of aggregating enrollment information for 
students receiving occupational education at BOCES centers 
and at local school districts falling within each BOCES region. 
Enrollment information which must be reported includes the 
numbers of students in each program by educational level, sex, 
ethnicity and type of student. In order to reduce the time 
and effort expended by BOCES personnel to fulfill enrollment 
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* However, past and present evaluation efforts at Nassau 
BOCES were also reviewed. 
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reporting requirements, enrollment reporting functions were 
included in the field trial* 

For secondary students, an attendance card file was 
already in use as part of the student registration process- 
This file contained each student's name, ID number, telephone 
number, local school name and code, grade level, and course 
name and code* 

SEED was used to enter the information in the card file 
into a computer data base. SEED was also used to enter the 
information from an existing teacher card file into the data 
base* Teacher information included in the data base consisted 
of each teacher's name and the names and code numbers for 
each teacher's courses* SEED was used to associate teachers 
with students and to generate a foria with the names and ID 
numbers of the students enrolled in each course* Next to 
each student's name, the teacher circled the appropriate 
identifiers for sex, ethnicity^ student type (general, disad- 
vantaged, handicapped) and work experience* After the forms 
were returned by the teachers, the data were key punched and 
merged with the other student data in the SEED-maintained 
data base* 

Since no card files existed for adult and special educa- 
tion students, forms were developed on which teachers entered 
student's names and the same enrollment information that was 
collect 2c for secondary students* Special education and adult 
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enrollment data were also keypunched and entered into the 
SEED-maintained data base. Special education, adult and 
secondary student information was then aggregated by SEED 
into a single matrix. The enrollment data were aggregated 
according to State Education Department enrollment reporting 
requirements. In addition, SEED was used to aggregate and 
report the enrollment information submitted to the BOCES by 
local school districts. 
B. Placement Information 

To obtain placement information, SEED would be used to 
create forms for graduating students which would be similar 
to the enrollment forms. Guidance counselors would encircle 
appropriate responses to placement questions appearing next 
to each student's name. Although placement data are re- 
ported by program, student sex, ethnicity, and so on, this 
information ordinarily would not have to be indicated by 
guidance counselors because it would have been entered into 
the data base at the time of the students' enrollment. 
However, guidance counselors could update such information 
as students ' addresses and telephone numbers in order to 
make follow-up studies more efficient. Placement data would 
be meryed into the data base and aggregated to meet report- 
ing requirements. 

These procedures were followed in the Nassau field trial 
with one exception. Computer forms with pre-entered student 
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names were not provided to guidance counselors.* However, 
for use during the past year only RRI and BOCES officials 
developed a new form for reporting placement data which 
reduced the efforts normally required of guidance personnel to 
report placement information. The data provided by guidance 
counselors were entered into the SEED data base and aggre- 
gated in accord with reporting requirements. SEED was then 
used to produce a placement report. Placement data sub- 
mitted to the BOCES by local school districts were also 
aggregated by SEED, and a placement report for these data was 
also generated. 
C . Follow-up Studies 

From the placement data, SEED generated lists of graduates, 
mailing labels, and decks of pre-punched cards for student 
graduates. The mailing labels were used to send follow-up 
survey forms to the graduates. (The mailings were followed 
up with phone calls when necessary.) When the forms were 
returned, the data were entered on the students* cards and 



* Placement data are reported for students who graduated in 
the prior academic year. These students had not yet been 
entered in the SEED-maintained data base. Thus, computer forms 
for guidance counselors could not be generated. However, 
students enrolled during the year that the field trial was 
conducted are included in the SEED-maintained data base. 
Therefore, during the forthcoming year, it will be possible 
to make use of the data base to produce placement forms for 
guidance counselors, and generally to make the reporting of 
placement information far more ef f icien''" . 
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merged into the student data base. After a sufficient propor- 
tion of the students had responded, SEED was used to aggregate 
the follow-up data in accord v;ith regional needs and state 
requirements. 

D. Summary of the Field Trial 

Nassau BOCES administrators indicated that even this 
limited use of the Reporting and Evaluation System resulted 
in a substantial saving of time and a reduction of the diffi- 
culties associated with meeting state reporting requirements. 
When the System is in full operation, even more reporting 
functions will be included. Further integration should result 
in even more savings of the time and effort of personnel. In 
addition, the processes set up for efficient reporting will 
also produce efficient program evaluation. 

Definite cost data were not compiled for the field trial 
since they would hardly be indicative of the cost of the 
System when it is in full operation. However, it was apparent 
to BOCES and RRI personnel that the integration of many oper- 
ating functions using a centralized reporting and evaluation 
system was exceptionally cost-effective. The time which 
occupational educational personnel were able to save was worth 
substantially more than the sum of EDP charges associated with 
data base operations and charges for the fulfillment of 
reporting requirements. 
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V, A Strategy for Statewide Implementation 
The Nassau field trial demonstrated that the Reporting 
and Evaluation System meets the system specifications set 
forth in Section III A of this report. Thus, it is reasonable 
to conclude that the System has been developed to the point 
where it is realistic to consider statewide implementation* 
Although it would not be possible to install the System at 
all locations in the immediate future, an incremental implemen 
tation strategy may be followed which will result in complete 
statewide implementation eventually. 

The strategy consists of adding system functions at the 
Nassau County BOCES (where the System is already installed) 
while simultaneously beginning to install the System at an 
increasing number of other locations. This approach permits 
the experience and capability gained at Nassau BOCES to be 
applied effectively to installation at other locations. 

The following tasks are recommended to the State Educa- 
tion Department in order to pursue this general approach to 
system implementation . 

A, Feasibility Study of the Implementation of the Reporting 
and Evaluation System in the Big-Six Cities 
The administration of occupational education in the large 
urban areas of the State is somewhat different than the admin- 
istration of occupational education in BOCES, local school 
districts or community colleges* Although the Reporting and 
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Evaluation System will be applicable to the needs of urban 
educational agencies, it may be necessary to adapt some of the 
operational details of the System to the more complex adminis- 
trative processes that exist in the cities. 
B. Adding System Functions at the Nassau BOCES 

The System is already installed at the Nassau County BOCES, 
and was used during this past year (in the field trial) to ful- 
fill state-mandated reporting requirements. During the 
forthcoming year, it would be desirable to increase the number 
of functions which the System can perform. 

The System should be augmented in ways which will encourage 
local school districts to participate in its use. For example, 
local districts may be interested in obtaining system-provided 
"state of occupational education reports" designed for the 
many local groups interested in publicly supported occupa- 
tional education. The report might include: 

• placement information; 

• follow-up reports, including reports on occupational 
graduates who have obtained job-related employment or 
who are participating in employer training programs; 

• enrollment reports, including reports of participation 
in occupational programs by each regional school 
district; 

• special reports on new programs and on programs which 
are coordinated with local-district program offerings; 
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• reports on adult education Programs; 

• reports on students in special education courses, 
broken down by student age and type of handicap; and 

• reports on the work experiences of occupational 
graduates . 

C . Installation of the System at New Locations 

The incremental implementation approach requires that the 
System be installed at several new locations during the forth- 
coming year. Since regional occupational educators seem to 
be concerned cbout reporting requirements, it would be sensi- 
ble to begin implementation in each new region with the 
system's capability to meet reporting requirements. As the 
demand for system services increases, the data base that is 
built for reporting purposes can be used to perform program 
evaluations and analyses, 

D. Procedures for Efficient System Implementation 

In order to bring about efficient system implementation, 
standard implementation procedures need to be developed and 
written materials, describing the System and its utilization, 
need to be developed. Specifically, the following products 
are considered by RRI to be sufficiently important to warrant 
development during the next year: 

1. A system guide * 

A simple guide to the Reporting and Evaluation 
System would be designed. This guide would illustrate what 
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the System does, explain the advantages to be expected from 
its use, and provide some guidelines for locally integrating 
currently discreet administrative processes (e.g., enrollment, 
attendance, placement, transcript production) now being per- 
formed at BOCES, community college, and school district loca- 
tions * 

2 . A user manual . 

Since many individuals would require information 
from the System's data base, the development of standard pro- 
cedures for using the System is an important component of 
system implementation. The manaal would include instructions 
for requesting data summaries, analyses, updates and so on. 
The manual would also contain instructions for filling out and 
filing a standard information request form with Management 
Control, 

E, Procedures for Inter-regional Communication 

Once the System is installed at two or more locations, 
experience in the use of the System and in the resolution of 
problems can be effectively consolidated only if there are 
formal communication channels, Tnus, in the initial stage 
of system implementation, the following should be developed: 
1. An information exchange ^ 

Experience with the System could be consolidated 
through an exchange of information in which participating 
personnel are informed of system developments, and different 
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ways of using the system to fulfill various reporting and 
evaluation tasks are shared* For example, when efficient 
methods are found for obtaining occupational enrollments from 
school districts and for entering this information in the 
System's data base, these methods should be shared with all 
locations using the System through an information exhange. 
The information exchange would also serve as a vehicle for 
informing local occupational educators of new capabilities 
and procedures at the management control level, 
2, Software exchange , 

A variety of software will be required to support 
local system functions. Much of this software already exists 
and is available. For example, standard scheduling and 
attendance packages are already in use at Nassau County BOCES, 
and would need little if any revision to be applied effectively 
at other locations. Thus, as the Reporting and Evaluation 
System is implemented, it would be useful to have regular 
bulletins produced describing programs under development, and 
containing up-to-date lists of available software. The bulletin 
should serve to eliminate duplicate efforts in software 
development. 

F. Subsequent Implementation 

After the communication products described in Sections 
Y D and TT E are developed, implementation can be carried out 
at a faster pace since the regions themselves will be able to 
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participate directly in implementation. Furthermore, after 
the feasibility study for implementation in Big-Six cities is 
completed, system use may be substantially expanded by instal- 
lation in New York City, or in other large urban areas not 
currently subscribing to the occupational services provided 
by the BOCES system. 
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